Egress Traffic Tracking and Stripe Billing Meters API Integration

This PR introduces an asynchronous solution for tracking egress traffic in the Freeway Gateway and reporting events to Stripe’s Billing Meters API.

RFC Reference

Storacha Network is implementing a scalable, automated mechanism for tracking egress traffic and updating Stripe’s API with relevant data to ensure accurate customer billing. This RFC outlines the proposed approaches and their trade-offs.

Implementation

The selected approach for this implementation is Alternative 3:

• Freeway Worker invoking usage/record, SQS, Lambda with Stripe Integration

Flow

1. To avoid blocking, the Freeway Gateway asynchronously invokes the usage/record capability using ctx.waitUntil.
2. A new handler added to w3infra/upload-api/stores/usage.js receives the events from the Kinesis stream and places the egress data into an Egress Traffic SQS queue.
3. A new Lambda function consumes the events from the SQS queue and publishes the egress data to the Stripe Billing Meters API.

graph TD
    CF[Freeway Gateway] --> w3infra[w3infra/upload-api]
    w3infra --> Kinesis[Kinesis Stream]
    Kinesis --> UsageRecord[Usage Record Handler]
    UsageRecord --> SQS[Egress Traffic Queue]
    SQS --> Lambda[Lambda Function]
    Lambda --> DynamoDB[Egress Traffic DynamoDB Table]
    Lambda --> Stripe[Stripe Billing Meters API]
    UsageRecord --> Logs[S3 Bucket /Receipts]

Key Advantages

• Asynchronous execution: ctx.waitUntil ensures the Freeway Worker won’t block, allowing seamless egress traffic handling.
• Decoupled architecture: By using SQS and Lambda, the architecture is decoupled and includes a buffer before publishing events to Stripe.
• Signed receipts: Each egress event generates a signed receipt, providing auditability and traceability. The receipts are stored in the general bucket in S3 where we already store other receipts.

Summary of Changes

• Introduces the usage/record capability handler in w3infra/upload-api/stores/usage.js to handle egress data from the Kinesis stream and place them into the SQS queue.
• Implements an event-driven architecture using SQS and Lambda to process egress events asynchronously and publish to Stripe.
• An integration test that publishes fake egress traffic events on Stripe’s Test API.

Next Steps

• Deployment (Staging)
• Validate the solution’s performance and scalability.
• Monitor for potential rate limit issues when publishing to Stripe.